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BACKGROUND:  Previous  studies  have  shown  that  certain 
orthopaedic  in  training  examination  scores  can  be  used  to 
identify  which  residents  may  be  at  risk  for  failing  the 
American  Board  of  Orthopaedic  Surgeons  (ABOS)  Part 
1  examination.  However,  no  studies  have  examined  how 
study  resources  may  affect  residents’  ABOS  Part  1  scores. 
The  goal  of  this  study  is  to  determine  which  review  sources 
or  review  courses,  if  any,  are  associated  with  improved 
ABOS  Part  1  scores. 

METHODS:  A  survey  was  sent  to  221  of  the  865  examinees 
who  took  the  ABOS  Part  1  examination  in  2012.  The 
questions  inquired  the  respondents  how  well  they  per 
formed  on  previous  orthopaedic  in  training  examinations 
and  ABOS  Part  1,  along  with  the  study  sources  they  most 
commonly  used,  review  courses  they  attended,  and  resour 
ces  they  would  recommended  if  they  were  to  retake  ABOS 
Part  1  examination. 

RESULTS:  Overall,  118  of  the  221  (53%)  survey  recipients 
completed  the  survey.  Six  (5%)  of  the  respondents  failed 
ABOS  Part  1  examination.  Orthobullets  and  the  American 
Academy  of  Orthopaedic  Surgeons  self  assessment  exami 
nations  were  recommended  as  the  primary  study  source 
significantly  more  (p  <  0.01)  than  most  other  resources, 
but  there  was  no  significant  association  between  study 
source  and  passing  ABOS  Part  1  or  scoring  in  a  certain 
percentile  on  ABOS  Part  1.  Similarly,  there  were  no 
associations  between  attending  a  review  course  and  either 
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passing  or  scoring  in  a  certain  percentile  for  ABOS  Part  1. 
Half  of  the  respondents  who  failed  ABOS  Part  1  attended 
multiple  review  courses. 

CONCLUSIONS:  There  does  not  appear  to  be  an  associ 
ation  between  improved  ABOS  Part  1  scores  and  orthope 
die  study  materials  or  review  courses.  Further  research  into 
the  value  of  certain  educational  modalities  should  be 
conducted  to  determine  the  best  ways  to  educate  orthopedic 
residents  and  determine  the  value  of  some  of  these 
commonly  used  orthopedic  review  modalities.  (J  Surg 
71:375  384.  Published  by  Elsevier  Inc.  on  behalf  of  the 
Association  of  Program  Directors  in  Surgery) 
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INTRODUCTION 

Much  like  the  many  studies  that  have  recently  been 
completed  regarding  the  orthopaedic  in  training  examination 
(OITE)fy11  many  articles  have  recently  examined  the 
associations  between  certain  performance  measures  and 
American  Board  of  Orthopaedic  Surgeons  (ABOS)  Part  1 
examination  scores. 11 '  The  ABOS  Part  1  examination  is  the 
first  part  of  the  board  certification  process  for  orthopedic 
surgeons  and  is  designed  to  evaluate  a  candidate’s  knowledge 
of  general  orthopedics  and  problem  solving  abilities.1  It  is 
necessary  for  orthopedic  residents  to  pass  this  exam  to 
become  board  eligible,  and  the  American  College  of  Grad 
uate  Medical  Education  has  mandated  that  each  residency 
program  maintain  a  pass  rate  of  75%  for  first  time  ABOS 
Part  1  examinees.16’18  Considering  its  importance  and  that 
the  failure  rate  for  ABOS  Part  1  is  much  higher  for  those 
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residents  who  failed  the  examination  on  their  first  attempt, 
it  is  easy  to  understand  why  there  is  significant  interest  in 
selecting  residents  who  are  felt  to  be  able  to  pass  this 
examination  and  determining  which  residents  are  most 
likely  at  risk  for  failing  it. 18 

However,  previous  studies  looking  at  various  factors 
associated  with  resident  performance  on  ABOS  Part  1  pass 
rates  do  not  appear  to  offer  information  on  what  study 
resources  have  been  used  by  residents  who  pass  ABOS  Part 
1  or  what  review  material  those  residents  would  recommend 
to  future  examinees.  Although  the  reasoning  behind  why 
some  residents  score  better  than  others  on  standardized 
examinations  is  likely  multifactorial,  ’  '  knowing  what, 
if  any,  associations  exist  between  resident  performance  and 
study  material  may  assist  program  directors  in  providing 
study  guidance  to  their  residents  for  ABOS  Part  1 .  It  is  not 
uncommon  for  residents  and  residency  programs  alike  to 
spend  thousands  of  dollars  on  studying  resources  or  review 
courses  in  hopes  of  being  better  prepared  to  pass  the  ABOS 
Part  1.  However,  to  the  author’s  knowledge,  there  have 
been  no  previous  studies  showing  that  specific  studying 
resources  or  review  courses  actually  improve  a  resident’s 
ability  to  pass  the  examination.  This  study  was  conceived  in 
hopes  of  determining  what  associations,  if  any,  exist 
between  ABOS  Part  1  performance  and  study  materials 
used  and  review  courses  attended. 

METHODS 

ABOS  Part  1  Examination 

The  ABOS  Part  1  examination  is  administered  by  the 
ABOS  and  is  the  capstone  examination  for  orthopedic 
training.  This  examination  is  the  first  of  the  2  examinations 
that  an  orthopedic  trainee  must  pass  to  become  board 
certified  by  the  ABOS.  Between  2001  and  2012,  77%  to 
89%  of  all  examinees  passed  ABOS  Part  1  on  their  first 
attempt.  1  Without  passing  Part  1,  the  candidate  is  ineli 
gible  to  sit  for  Part  2.  Most  residents  sit  to  take  ABOS  Part 
1  immediately  after  the  completion  of  their  orthopedic 
residencies.  Although  there  is  no  limit  on  the  number  of 
times  an  applicant  can  sit  for  Part  1,  data  suggest  that  those 
participants  who  fail  once  are  at  a  much  higher  risk  of 
failing  again  on  repeat  examination.  ~  The  American 
College  of  Graduate  Medical  Education  Orthopaedic  Sur 
gery  Residency  Review  Committee  requires  that  all  accred 
ited  programs  maintain  a  first  time  pass  rate  of  at  least  75% 
for  all  residents  for  this  examination. 

Examination  Preparation 

There  is  no  formalized  curriculum  to  prepare  for  ABOS 
Part  1.  Many  residents  attend  review  courses,  complete 
practice  questions  from  previous  OITEs  or  practice  exami 
nations,  and  read  review  books.  In  general,  most  residents 


seem  to  use  a  combination  of  all  3  methods  as  they  prepare 
for  the  examination.  Commonly  attended  review  courses  are 
as  follows:  Miller’s  review  course,  Maine  orthopaedic  review 
course,  and  American  Academy  of  Orthopaedic  Surgeons 
(AAOS)  orthopaedic  board  preparation  and  review  course. 
Other  study  materials  that  are  also  commonly  used  include 
the  following:  Orthobullets  (www.orthobullets.com),  AAOS 
comprehensive  orthopaedic  review  (ed,  Lieberman  JR), 
AAOS  self  assessment  examinations,  AAOS  orthopaedic 
knowledge  update  10  (ed,  Flynn  JM),  AAOS  orthoportal 
(ed,  Sarwark  J),  previous  years’  OITE  questions,  and 
Miller’s  review  of  orthopaedics  (ed,  Miller  MD). 

Survey  of  Examinees 

The  only  organization  that  has  ABOS  Part  1  results  and 
contact  information  for  all  of  the  examinees  is  the  ABOS, 
and  such  material  is  their  propriety  information.  As  such,  a 
survey  was  sent  via  e  mail  to  221  of  the  865  examinees 
(26%)  of  the  2012  ABOS  Part  1  examination  who  partici 
pated  in  the  2012  online  Miller/Orthobullets  virtual  review 
curriculum.  This  was  a  free,  online  review  curriculum 
developed  for  ABOS  Part  1  review.  An  e  mail  was  sent  to 
all  of  the  participants  in  this  curriculum  offering  them 
inclusion  in  this  study.  Although  aimed  at  senior  level 
orthopedic  residents,  attending  surgeons  who  had  previously 
taken  ABOS  Part  1  and  failed  or  international  graduates  who 
were  studying  to  take  ABOS  Part  1  could  also  participate  in 
the  Miller/Orthobullets  review.  Respondents  were  asked  to 
complete  a  survey  regarding  their  study  habits  and  examina 
tion  performance  as  completely  as  possible.  As  can  be  seen  by 
analyzing  the  survey  (Appendix  A),  the  questions  were  meant 
to  delineate  how  well  the  respondent  performed  on  previous 
OITEs  and  ABOS  Part  1 ,  along  with  the  study  sources  most 
commonly  used,  review  courses  attended,  and  resources 
recommended  for  the  ABOS  Part  1  examination  if  they 
were  to  repeat  the  examination.  The  survey  was  hosted  on 
surveymonkey.com,  and  response  data  were  downloaded 
direcdy  from  the  website. 

Statistical  Analysis 

Fisher  exact  test  was  used  to  determine  which  responses 
were  associated  with  passing  and  scoring  in  higher  percen 
tiles  for  the  ABOS  Part  1  examination.  For  question  8,  there 
were  10  study  methods  listed  as  tools  to  determine  which 
variables  were  ranked  the  highest  as  study  tools  for  the 
examination.  There  were  3  ways  in  which  study  methods 
were  compared  with  each  other.  First,  it  was  determined 
whether  or  not  the  study  was  ranked  first,  as  the  primary 
study  method.  Second,  it  was  determined  if  the  study  tool 
was  ranked  first,  second,  or  third  (it  was  at  least  used 
somewhat  for  studying)  compared  with  not  ranked  at  all 
(the  respondent  did  not  use  that  particular  resource  for 
studying).  Third,  each  study  method  was  compared  after 
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assigning  them  weighted  scores  calculated  by  squaring  the 
rank  of  the  particular  method.  This  weighted  score  used  was 
based  on  the  assumption  that  a  person  would  be  more  likely 
to  use  their  primary  recommended  resource  substantially 
more  than  a  source  they  recommend  as  the  third  choice.  If 
the  method  was  ranked  first,  it  received  a  weighted  score  of 
1.  If  it  was  ranked  second,  it  received  a  weighted  score  of  4. 
If  it  was  ranked  third,  it  received  a  weighted  score  of  9,  and 
if  it  was  not  ranked  at  all,  it  received  a  weighted  score  of  16. 
The  means  of  the  weighted  scores  were  compared  using 
Student  t  test  with  a  Dunnett  correction. 


RESULTS 

Overall,  118  (53%)  of  the  221  survey  recipients  completed 
the  survey.  Six  (5%)  respondents  reported  failing  ABOS 
Part  1.  The  percentiles  reported  for  each  respondents’ 
ABOS  Part  1  and  OITE  can  be  seen  in  Table  1.  No  more 
than  10%  of  the  respondents  scored  above  the  90th 
percentile  on  either  of  the  postgraduate  year  (PGY)  OITEs 
or  ABOS  Part  1.  The  median  percentile  range  for  each 
OITE  was  slightly  higher  than  the  median  percentile  for 
ABOS  Part  1.  No  respondent  who  scored  above  the  70th 
percentile  on  their  PGY  4  or  PGY  5  OITE  failed  ABOS 
Part  1. 

An  analysis  of  the  OITE  scores  for  those  respondents 
who  passed  and  those  who  failed  ABOS  Part  1  can  be  seen 
in  Figure  1.  There  was  no  significant  association  between 
percentile  scores  for  PGY  3  (p  =  0.1443),  PGY  4  (p  = 
0.7866),  or  PGY  5  (p  =  0.0762)  OITE  scores  and  passing 
ABOS  Part  1.  However,  as  Figure  2  demonstrates,  respond 
ents  who  failed  were  significantly  (p  =  0.0475)  more  likely 
to  score  in  the  25  th  percentile  or  lower  during  their  PGY 
5  OITE  than  those  who  passed  (3/6  [50%]  vs  19/110 
[17%],  respectively).  The  overall  relationship  between 
OITE  percentile  scores  and  ABOS  Part  1  percentile  scores 
can  be  better  appreciated  in  Figure  3.  For  all  PGYs,  there 
was  a  significant  (PGY  3,  p  <  0.0001;  PGY  4,  p  =  0.0169; 
and  PGY  5,  p  =  0.005)  association  between  a  respondent’s 
score  on  the  OITE  and  their  ABOS  Part  1  percentile  scores. 
Respondents  tended  to  score  in  the  same  percentile  range 
on  ABOS  Part  1  as  they  had  scored  on  the  OITE. 

The  primary  study  sources  the  respondents  used 
for  ABOS  Part  1  can  be  seen  in  Table  2.  The  AAOS 


self  assessment  examinations  (38/1 16,  32%)  and  Orthobul 
lets  (37/116,  32%)  were  listed  as  the  primary  study  source 
of  the  respondents  significantly  (p  <  0.01)  more  than  all 
other  resources  except  for  Miller’s  review  of  orthopaedics 
(28/116,  24%).  In  fact,  the  AAOS  self  assessment  exami 
nations,  Orthobullets,  and  Miller’s  review  of  orthopaedics 
were  chosen  as  the  primary  study  source  more  than  4  times 
as  often  as  all  other  resources.  Still,  there  was  no  association 
found  between  using  a  specific  primary  study  source  and 
scoring  in  a  particular  percentile  or  passing  ABOS  Part  1 
(p  =  0.0509  and  p  =  0.2314,  respectively)  (Fig.  4). 
However,  5  of  the  10  respondents  who  scored  above  the 
90th  percentile  on  ABOS  Part  1  indicated  that  their 
primary  study  source  were  the  AAOS  self  assessment 
examinations.  When  ranking  which  study  resources  the 
respondents  would  use  if  they  were  to  take  ABOS  Part 
1  again,  more  participants  indicated  that  they  would  use  the 
AAOS  self  assessment  examinations  as  their  primary 
resource  (41/116,  35%)  than  any  other  source.  The  only 
other  resources  that  at  least  10  of  the  respondents  stated 
they  would  use  as  their  primary  study  source  if  they  were  to 
retake  ABOS  Part  1  were  Orthobullets  (31/116,  27%), 
Miller’s  review  of  orthopaedics  (19/116,  16%),  and  content 
from  the  Miller’s  orthopaedic  review  course  (11/116,  9%). 
Orthobullets  was  ranked  as  the  primary,  secondary,  or 
tertiary  study  source  significantly  more  often  (86/116, 
75%;  p  =  0.0346)  than  all  other  resources,  and  the  mean 
weighted  score  of  each  study  source  recommended  by  the 
respondents  can  be  seen  in  Figure  5.  This  figure  demon 
strates  that  when  looking  at  all  of  the  recommendations 
from  respondents  as  to  if  they  would  use  a  particular 
resource  as  a  primary,  secondary,  or  tertiary  study  source 
if  they  were  to  take  ABOS  Part  1  again,  Orthobullets  and 
the  AAOS  self  assessment  examinations  were  given  higher 
recommendations  than  all  other  study  materials. 

Only  8%  (9/1 16)  of  respondents  stated  that  they  did  not 
attend  a  review  course  before  taking  ABOS  Part  1 ,  and  1 8% 
(21/116)  of  all  respondents  stated  that  they  went  to  more 
than  1  review  course.  Half  of  the  respondents  who  failed 
went  to  more  than  1  review  course,  whereas  none  of  the 
respondents  who  did  not  go  to  a  review  course  failed 
(Table  3).  There  were  no  significant  differences  found 
between  passing  and  failing  regarding  which  review  courses 
were  attended  except  for  the  Maine  orthopaedic  review 
course.  Half  of  the  respondents  who  failed  (3/6)  attended 


TABLE  1.  The  Number  of  Respondents  Who  Scored  Within  Each  Percentile  Range  (Percentages  in  Parentheses) 


Percentile  Scored 

PGY  3  OITE 

PGY  4  OITE 

PGY  5  OITE 

ABOS  Part  1 

Less  than  26 

10 

9% 

16 

14% 

22 

19% 

25 

21% 

26-50 

33 

29% 

39 

35% 

34 

30% 

35 

30% 

51-70 

40 

36% 

31 

27% 

25 

22% 

28 

24% 

71-90 

23 

21% 

20 

18% 

22 

19% 

20 

17% 

Greater  than  90 

6 

5% 

7 

6% 

12 

10% 

10 

8% 

Median  percentile 

51-70 

51-70 

51-70 

26-50 

Total  respondents 

112 

113 

118 

118 
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□  ABOS  Part  1  Fail  E  ABOS  Part  1  Pass 


Orthopaedic  In-Training  Examination  scores  by  Post-Graduate  Year 


FIGURE  1 .  Graph  showing  the  percentage  of  respondents  who  passed  or  failed  the  ABOS  Part  1  examination  and  what  their  percentile  scores  were 
on  the  PGY  3,  4,  and  5  OITEs. 


that  course,  whereas  only  10%  (11/110)  of  the  respondents 
who  passed  the  examination  attended  that  course  (p  = 
0.0030).  However,  it  should  be  noted  that  all  of  the 


respondents  who  went  to  the  Maine  orthopaedic  review 
course  and  subsequently  failed  also  went  to  at  least  1  other 
review  course.  Figure  6  shows  that  there  was  little  difference 
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0% 
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□  Pass 
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6% 

31% 

35% 
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14% 

6% 
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19% 
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11% 
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21% 

20% 

OITE  percentile  scores  for  each  PGY  listed 


FIGURE  2.  Graph  showing  the  percentage  of  respondents  who  scored  in  each  specified  percentile  range  on  the  ABOS  Part  1  examination  and  what 
their  percentile  scores  were  on  the  PGY  3,  4,  and  5  OITEs.  #:  significant  difference  between  those  who  passed  and  failed  ABOS  Parti . 
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FIGURE  3.  Graph  showing  the  percentage  of  respondents  who  ended  up  scoring  within  a  specific  percentile  range  on  the  ABOS  Part  1  examination 
after  scoring  a  specific  percentile  on  their  PGY  3,  4,  and  5  OITEs. 


in  percentile  scores  for  both  OITE  and  the  ABOS  Part 
1  scores  between  those  respondents  who  attended  multiple 
review  courses  and  those  who  did  not  go  to  a  review  course. 
When  looking  at  the  PGY  4  and  PGY  5  OITE  percentile 
scores,  it  was  found  that  40%  (4/10)  of  respondents  who 
did  not  go  to  a  review  course  stated  that  their  PGY  4  and 
PGY  5  OITE  scores  were  in  the  25th  percentile  or  lower 
compared  with  5%  (1/21)  and  20%  (4/21)  of  those  who 
went  to  multiple  review  courses,  respectively. 

DISCUSSION 

Passing  the  ABOS  Part  1  examination  is  necessary  to  become 
a  board  certified  orthopedic  surgeon.  Although  the  overall 
pass  rate  for  ABOS  Part  1  improved  from  79%  in  2011  to 


TABLE  2.  The  Number  of  Respondents  Who  Listed  Each 
Resource  as  Their  Primary  Studying  Source  For  ABOS  Part  1 


Primary  Study  Source  for  ABOS  Part  1 


AAOS  self-assessment  exam  questions 

38 

32% 

Orthobullets  website 

37 

31% 

Miller's  review  of  orthopaedics  textbook 

28 

24% 

AAOS  comprehensive  orthopaedic  review 
Previous  years'  OITE  questions 

6 

5% 

4 

3% 

Other 

4 

3% 

AAOS  orthopaedic  knowledge  update 

1 

1% 

85%  in  2012,  a  15%  failure  rate  means  that  130  of  the  865 
trainees  who  sat  for  the  examination  in  2012  failed. 
Learning  what  studying  materials  or  review  courses  may 
improve  a  candidate’s  ability  to  score  well  on  ABOS  Part  1, 
if  any,  could  be  extremely  valuable  in  helping  residents 
improve  their  chances  of  passing  the  examination. 

The  respondents  in  this  study  had  a  lower  failure  rate 
(5%)  than  the  overall  failure  rate  for  all  takers  of  ABOS 
Part  1  in  2012  but  had  a  failure  rate  that  was  very  close  to 
that  of  first  time  US  and  Canadian  test  takers  (6%). 
Although  the  low  number  of  failures  made  it  difficult  to 
determine  significance  regarding  passing  or  failing  ABOS 
Part  1,  this  study  clearly  demonstrates  that  improved 
percentile  scores  on  PGY  3,  4,  and  5  OITE  examinations 
are  associated  with  improved  percentile  scores  on  ABOS 
Part  1.  This  finding  is  in  line  with  numerous  other  studies 
that  have  shown  an  association  or  a  correlation  between 
these  scores1  '  '  ’  and  is  also  the  first  to  show  such 
associations  when  looking  at  residents  who  took  both  the 
OITE  and  the  A  BOS  Part  1  using  computerized  examina 
tions.  ’’  More  importantly,  these  findings  help  demon 
strate  that  the  responses  used  in  this  study  came  from 
examinees  who  were  relatively  diverse  in  terms  of  scoring  on 
both  the  OITE  and  ABOS  Part  1.  This  variety  of  high  and 
low  scores  on  the  OITE  and  ABOS  Part  1  is  likely  to  be 
present  in  many  residencies,  helping  increase  the  external 
validity  of  this  study’s  results. 
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FIGURE  4.  Graph  showing  the  percentage  of  respondents  who  scored  within  a  specific  percentile  range  on  the  ABOS  Part  1  examination  and  what 
their  primary  study  source  was  during  examination  preparation.  AAOS  COR,  American  Academy  of  Orthopaedic  Surgeons  comprehensive 
orthopaedic  review;  AAOS  OKU- 10,  American  Academy  of  Orthopaedic  Surgeons  orthopaedic  knowledge  update  10;  AAOS  SAE,  American 
Academy  of  Orthopaedic  Surgeons  self-assessment  examinations;  MRO,  Miller's  review  of  orthopaedics;  OB,  www.orthobullets.com. 


This  study  found  no  associations  between  using  a  specific 
study  source  and  either  scoring  within  a  particular  percentile 
or  passing  ABOS  Part  1.  Still,  there  were  some  general 
patterns  that  are  worth  mentioning.  First,  respondents  used 
the  AAOS  self  assessment  examinations  and  Orthobullets  as 
their  primary  studying  source  significantly  more  than  any 
other  source  except  for  Miller’s  review  of  orthopaedics. 
Using  previous  OITE  questions  has  been  previously  deter 
mined  to  correlate  with  improved  OITE  scores,2  and  this 
study  shows  that  using  question  based  sources  seems  to  be 
the  preferred  method  of  review  for  this  cohort.  It  seems 
likely  that  trainees  would  like  to  try  to  maximize  their 
exposure  to  questions  before  taking  ABOS  Part  1  for  a 
couple  of  reasons.  First,  it  helps  to  prepare  the  examinee  for 
the  format  of  the  examination.  Second,  it  appears  that  some 
orthopedic  topics  are  either  deemed  more  important  to 
understand  or  easier  to  test  than  others.  Practice  questions, 
regardless  of  source,  seem  to  be  more  likely  to  expose 
examinees  to  these  topics  during  their  studying  than  study 
ing  formats.  Lastly,  it  is  important  to  note  that  this  cohort 
of  responders  was  derived  from  an  online  review  course  run 
by  a  company  (Orthobullets)  that  is  largely  popular  because 
of  the  questions  it  provides  to  its  users.  Therefore,  this 
cohort  may  be  more  likely  to  favor  questions  over  other 
forms  of  review  materials  (i.e.,  books)  based  on  sampling 
bias  alone.  Still,  it  is  worth  noting  that  the  respondents  of 


this  study,  of  which  94%  passed,  stated  that  they  would 
recommend  using  the  question  based  AAOS  self  assessment 
examinations  and  Orthobullets  more  than  any  other 
resource  if  they  were  to  prepare  for  ABOS  Part  1  again. 
Such  a  finding  seems  to  indicate  that  these  respondents  felt 
that  these  2  studying  resources  prepared  them  well  for  the 
examination. 

The  findings  of  this  study  in  terms  of  ABOS  Part 
1  performance  and  participation  in  an  orthopedic  review 
course  are  interesting.  Overall,  there  were  no  associations 
found  between  scoring  in  a  particular  percentile  or  passing 
ABOS  Part  1  and  attending  a  specific  review  course  except 
for  the  Maine  orthopaedic  review  course.  Although  the 
respondents  who  failed  were  significantly  more  likely  to  go 
to  the  Maine  orthopaedic  review  course,  it  should  be 
highlighted  that  those  respondents  who  went  to  that  review 
course  and  failed  went  to  at  least  one  other  review  course  as 
well.  When  looking  at  the  respondents  who  represented  the 
extremes  of  review  course  attendance  (those  who  did  not  go 
to  a  review  course  and  those  who  went  to  more  than  1),  it  is 
noted  that  there  were  very  little  difference  between  their 
ABOS  Part  1  percentiles.  These  findings  may  reflect  more 
about  the  participants  attending  (or  not  attending)  a 
particular  course  than  the  courses  themselves,  as  those 
who  attended  a  course  may  have  felt  that  they  were  not  as 
prepared  for  the  ABOS  Part  1  from  their  residency,  had 
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FIGURE  5.  Graph  showing  the  weighted,  mean  rank  of  each  review  source  listed.  OB,  www.orthobullets.com;  AAOS  SAE,  American  Academy  of 
Orthopaedic  Surgeons  self-assessment  examinations;  MRO,  Miller's  review  of  orthopaedics;  MRC  Content,  content  obtained  from  Miller's  review 
course;  AAOS  COR,  American  Academy  of  Orthopaedic  Surgeons  comprehensive  orthopaedic  review;  ARC  content,  content  obtained  from  the 
American  Academy  of  Orthopaedic  Surgeons  orthopaedic  board  preparation  and  review  course;  Maine  RC  content,  content  obtained  from  the  Maine 
orthopaedic  review  course;  AAOS  OKU-10,  American  Academy  of  Orthopaedic  Surgeons  orthopaedic  knowledge  update-  10;  AAOS  orthoportal, 
http:/  /  orthoportal.aaos.org/. 


more  anxiety  about  the  examination,  or  studied  less 
independently  because  they  attended  at  least  1  review  course 
than  those  who  chose  not  to  attended  such  a  course. 
However,  it  is  worth  noting  that  40%  of  the  respondents 


TABLE  3.  The  Number  of  Respondents  Who  Went  to  Each 
Listed  Review  Course  or  Combination  of  Review  Courses  and 
Those  Review  Courses  That  Those  Respondents  Who  Failed 
ABOS  Part  1  Attended. 


Review  Course 

Participants 

Failed  (%) 

AAOS  only 

10 

0 

0 

MRC  only 

73 

3 

4 

Maine  only 

2 

0 

0 

None 

10 

0 

0 

Review  course  not  listed 

1 

0 

0 

AAOS  and  MRC 

9 

0 

0 

AAOS  and  Maine 

3 

1 

33 

MRC  and  Maine 

7 

1 

14 

All  3  courses 

2 

1 

50 

MRC,  Miller's  review  course; 

Maine,  Maine 

orthopaedic 

review 

course. 

Of  6,  3  (50%)  respondents  who  failed  went  to  multiple  review  courses. 
Of  10,  2  (20%)  respondents  who  scored  above  90th  percentile  on 
ABOS  Part  1  went  to  multiple  courses. 

Of  25,  7  (28%)  respondents  who  scored  less  than  25th  percentile  on 
ABOS  Part  1  went  to  multiple  review  courses. 


who  stated  they  did  not  attend  a  review  course  scored  in  the 
25th  percentile  or  lower  during  their  PGY  4  and  PGY 
5  OITEs  compared  with  5%  and  20%  of  those  who 
attended  multiple  review  courses,  respectively.  Such  a 
finding  would  seem  to  suggest  that  those  the  40%  of 
respondents  who  did  not  attend  a  review  course  and  scored 
below  the  25th  percentile  on  their  PGY  5  OITE  would  be 
more  concerned  about  their  potential  for  failing  ABOS 
Part  1  than  5  and  20%  of  respondents  who  attended 
multiple  courses  and  scored  below  the  25th  percentile  on 
their  PGY  4  and  PGY  5  OITEs,  respectively.  This  would 
seem  especially  true  considering  that  the  largest  previous 
study  looking  at  correlations  between  OITE  and  ABOS  Part 
1  scores  found  that  the  correlation  between  OITE  score  and 
ABOS  Part  1  result  increased  with  resident  matriculation 
and  that  probability  of  passing  ABOS  Part  1  was  only  below 
90%  for  those  residents  who  scored  below  the  20th 
percentile  on  their  PGY  4  and  PGY  5  OITEs.  Still,  with 
such  a  low  number  of  responses  from  members  of  both 
groups,  these  data  remain  fragile  and  should  not  be  used  to 
draw  strong  conclusions  regarding  the  value  or  effectiveness 
of  review  courses. 

This  study  is  not  without  limitations.  Despite  its  rela 
tively  large  size  in  comparison  with  previous  studies  on  this 
topic,  only  14%  of  all  examinees  for  ABOS  Part  1  in  2012 
participated  in  this  survey.  Respondents  who  typically  do 
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FIGURE  6.  Graph  showing  the  specified  percentiles  the  respondents  scored  on  PGY  3,  4,  and  5  OITEs  and  ABOS  Part  1  if  they  had  gone  to  multiple 
review  courses  or  no  review  courses. 


well  or  have  enjoyed  the  subject  being  discussed  commonly 
bias  surveys  by  responding  at  a  greater  rate  than  those  who 
have  had  a  negative  experience.  This  may  explain  why  the 
pass  rate  for  our  respondents  was  higher  than  what  would  be 
expected  based  on  the  data  regarding  pass  rates  obtained 
from  the  ABOS.  However,  our  pass  rate  was  very  similar 
to  that  of  first  time  US  and  Canadian  test  takers.  This  is 
likely  because  any  international  graduate  sitting  for  ABOS 
Part  1  would  not  have  taken  the  OITE  and  would  be 
unable  to  fully  participate  in  the  survey.  Additionally,  only 
those  people  who  completed  the  2012  Miller/Orthobullets 
virtual  review  curriculum  received  a  survey,  making  this  a 
convenience  sample.  This  limitation  therefore  means  the 
respondents  may  not  adequately  represent  the  general 
population  of  test  takers  and  may  decrease  the  external 
validity  of  this  study.  However,  this  limitation  also  means 
that  this  cohort  represents  a  group  of  examinees  who  are 
taking  both  the  OITE  and  ABOS  Part  1  in  a  computerized 
format.  Such  a  cohort  has  not  been  evaluated  by  previous 
studies  looking  at  OITE  and  ABOS  Part  1  correlations,  '^’lh 
and  it  is  likely  that  the  role  of  online  review  courses  and 
materials  would  only  increase  in  the  future.  Further 
more,  the  survey  was  not  limited  to  first  time  test  takers 
or  to  US  and  Canadian  candidates.  It  is  unknown  how 
many  of  the  respondents  had  previously  taken  ABOS 
Part  1  and  failed.  If  all  6  of  the  respondents  from  the 


survey  had  already  failed  once  before,  this  could  dramati 
cally  affect  the  external  validity  of  these  results.  However, 
the  distribution  of  this  survey  certainly  provided  us  with  a 
wide  variety  of  residents  from  multiple  programs  unlike 
previous  studies,1  ’  and  it  appears  that  only  the  ABOS 
themselves  would  have  the  ability  to  gather  similar  data  on  a 
larger  group  of  examinees.  Surveys  are  also  based  on  recall 
and  self  reporting,  both  of  which  carry  their  own  inherent 
biases.  There  was  no  way  for  the  authors  to  validate  the 
respondents’  scores,  as  such  information  is  proprietary 
information  of  the  ABOS.  Lastly,  we  do  not  know  how 
many  of  these  respondents  had  taken  the  examination 
before  and  how  many  were  taking  it  for  the  first  time.  It 
may  be  that  repeat  test  takers  have  much  different  study 
habits  than  first  time  test  takers. 


CONCLUSION 

Considering  the  devotion  to  evidence  based  medicine 
principles,  it  seems  intuitive  that  similar  practices  should 
be  sought  for  orthopedic  education.  Despite  the  limitations 
mentioned,  this  study  appears  to  be  the  first  to  evaluate  the 
effect  of  both  specific  study  materials  and  review  courses  on 
ABOS  Part  1  performance.  Although  there  were  no 
significant  associations  found  between  specific  studying 
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materials  or  review  courses  and  passing  or  scoring  in  certain 
percentiles  in  the  ABOS  Part  1  examination,  this  study 
seems  to  indicate  that  question  based  review  sources  are 
preferred  over  books.  However,  considering  the  limited 
sample  size  and  limitations  mentioned  within  this  article,  it 
is  impossible  to  say  that  no  such  relationship  exist.  Until 
further  studies  can  be  completed  to  further  address  what 
studying  materials,  if  any,  are  correlated  with  improved 
ABOS  Part  1  scores  or  pass  rates,  residency  directors  and 
residents  alike  should  continue  to  use  whatever  study 
resources  they  feel  they  learn  best  from.  The  results  of  this 
study  should  be  used  to  further  guide  prospective  studies 
that  examine  the  value  of  certain  studying  resources  and  not 
as  definitive  evidence  stating  which  resource  are  or  are  not 
valuable. 
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